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9.1.10 Solved Problems 

Problem 1 

LetX ~ N( 0, 1). Suppose that we know 

Y \X=x ~ N(x, 1). 

Show that the posterior density of X given Y = y,f X \ yi x >’)•> is given by 


X\ Y = y 



• Solution 

y i 

o Our goal is to show that f x ^ y(x \y) is normal with mean - and variance 
Therefore, it suffices to show that 


f X \ y( x It) = c-(v)cxp < - IV 


y 


where c(y) is just a function of y. That is, for a given y, c(y) is just the normalizing 
constant ensuring that f x ^ | y) integrates to one. By the assumptions, 


1 | (y-xY 

fy\rfy \ x ) = ^ exp ' 


1 I x“ 
f/ x ) = l^= ex P' 

A/27T 


Therefore, 
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f Y \x(y\ x ')M x ') 

f ^ xM= m 

= (a function of y) ■ f Y \ X (y\x)f x (x) 


= (a function of y) ■ exp 


= (a function of y) ■ exp 


= (a function of y) ■ exp 


(y - x) 2 + x 2 | 



Problem 2 

We can generalize the result of Problem 9.1 using the same method. In particular, assuming 

X ~ N(ju, t 2 ) and Y \ X = x ~ N(x, a 2 ), 
it can be shown that the posterior density of X given Y = v is given by 


X \ Y = y 


/ y/a 2 +ju/z 2 1 

\yl/a 2 + Hr 2, 1 /cr 2 + 1 /t 2 


In this problem, you can use the above result. LetX ~ N(ju, z 2 ) and 

Y | X= x ~ N(x, a 2 ). 

Suppose that we have observed the random sample 7j, Y 2 , •••, Y such that the P.'s are i.i.d. 
and have the same distribution as Y. 


a. Show that the posterior density of X given Y (the sample mean) is 


X I Y 


nY/a 2 + fil x 2 
nt a 2 + 1/t 2 

\ 


1 


\ 


n/a 2 + 1/z 2 J 
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b. Find the MAP and the MMSE estimates of X given Y. 

• Solution 

o a. Since Y \ X = x ~ N(x, a 2 ), we conclude 


Y \X=x 


Nix, 


Therefore, we can use the posterior density given in the problem statement 

2 

(we need to replace cr by —). Thus, the posterior density of X given Y is 


/ 


X I Y 


N 


nY/a 2 + /i/z 2 


1 


n! a 1 + 1 / t z n! a~ + 1 It 1 


b. To find the MAP estimate of X given Y, we need to find the value that 
maximizes the posterior density. Since the posterior density is nonnal, the 
maximum value is obtained at the mean which is 


X MAP 


nY/a 2 + fill 2 
nlo 2 + 1/r 2 


Also, the MMSE estimate of X given Y is 


X m =E[X\Y} 


nY/a 2 + fi/z 2 
n! a 2 + 1 / z 2 


Problem 3 

Let X M be the MMSE estimate of X given Y. Show that the MSE of this estimator is 

MSE = £[Var(X| T)]. 


• Solution 

o We have 
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Var(X| Y) = E[(X - E[X\ 7])~ | Y] (by definition of Var(X| 7)) 

= E[(X-X m ) 2 \Y]. 

Therefore, 

E[Var(X\Y)] = E[E[(X-X M )\Y]] 

= E[(X- X M ) 2 ] (by the law of iterated expectations) 

= MSE (by definition of MSE). 


Problem 4 

Consider two random variables X and Y with the joint PMF given in Table 9.1. 



a. Find the linear MMSE estimator of X given Y, (X L ). 

b. Find the MMSE estimator of X given Y, (X M ). 

c. Find the MSE ofX M . 

• Solution 

o Using the table we find out 


P^) 

pm 

Pyi 0) 
^y(l) 
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2 

Thus, the marginal distributions ofXand 7are both Bernoulli (j). Therefore, we 
have 

2 

EX - EY = -, 

Var W - Var(7) - §• f- |. 

a. To find the linear MMSE estimator of X given Y, we also need Cov(X, 7). 
We have 


EXY= Yj x iyj p xy(xx) = 0. 

Therefore, 


Cov(X, 7) = EXY- EXEY 
4 

~ ~ 25' 


The linear MMSE estimator of X given 7 is 




Cov(X, 7) 
Var(7) 


( Y-EY) + EX 


-4/25 



2 

5 


Since 7 can only take two values, we can summarize X L in the following 
table. 


Table 9.2: The linear MMSE estimator ofXgiven 7for Problem 4 



II 

o 

Y= 1 


2 



3 

0 


b. To find the MMSE estimator ofXgiven 7, we need the conditional PMFs. 
We have 
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Thus, 


p x\ y(° I °) _ 

1 

5 1 

“ I “ 3‘ 

5 


PxA o, o) 

p y(0) 


P x\y^ 1 0 ) = 1 — ~ |- 


We conclude 


X\Y= 0 


Bernoulli 



Similarly, we find 

p x\ y(0 1 1) = 1, 

Wi|i) = o. 

Thus, given 7=1, we have always X = 0. The MMSE estimator of X given 
7 is 


X m = E[X\Y]. 


We have 


2 

E\X\ 7=0]= - 
E[X\Y= 1] = 0. 


Thus, we can summarize X M in the following table. 


Table 9.3: The MMSE estimator ofX 


given 7 for Problem Problem 4 



o 

II 

II 

X M 

2 

3 

0 
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We notice that, for this problem, the MMSE and the linear MMSE 
estimators are the same. In fact, this is not surprising since here, Y can only 
take two possible values, and for each value we have a corresponding 
MMSE estimator. The linear MMSE estimator is just the line passing 
through the two resulting points. 

c. The MSE ofX M can be obtained as 


MSE = EiX 2 ] 

= EX 2 -E[X m \ 

2 „ 2 

= - 5 ~e[x m I 

,2 4 

From the table for X M , we obtain E[X M ] = —. Therefore, 

mse =Ts■ 

Note that here the MMSE and the linear MMSE estimators are equal, so 
they have the same MSE. Thus, we can use the fonnula for the MSE of X L 
as well: 


MSE = (1 -p(X, Y) 2 )Var(X) 

1 Cov(X,7) 2 \ 

“ y Var(X)Var(Y) j Var(Z) 

( - 4/25) 2 \ 6 
6/25 • 6/25 125 

2 

15' 



Problem 5 

Consider Example 9.9 in which X is an unobserved random variable with EX = 0, VarfY) = 4. 
Assume that we have observed Y { and Y 1 given by 

Y\ =X+W V 

y 2 =x+w 2 , 
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where EW X = EW 2 = 0, Var (JV X ) = 1, and Var( W 2 ) = 4. Assume that W x , W 2 , and A are 
independent random variables. Find the linear MMSE estimator of X given F| and Y 1 using 
the vector formula 


± l = C xy C y ~ * 1 * (Y-E[Y]) + E[X]. 


• Solution 

o Note that, here, Xis a one dimensional vector, and Y is a two dimensional vector 


We have 


Y = 


'V 


m X+ w x ' 

^2 


x+w 2 


' Va^Tj) 

Cov(7 1? Y 2 ) ■ 


'5 4‘ 

Cov(7 2 , Tj) 

Var(7 2 ) 


4 8 


Therefore, 


C XY = [ Cov(X ’ Cov(X ’ 7 2) ] = [4 4 ]. 


* z =[ 4 4 ] 


'5 

4' 


' 7j 


■O' 

\ 

4 

8 

l 

^2 


0 

) 


+ 0 


2 1 
3 6 


7, 


To 


2 1 
~ 3 7l + 6 Yl ’ 


which is the same as the result that we obtained using the orthogonality principle 
in Example 9.9 . 


Problem 6 

Suppose that we need to decide between two opposing hypotheses // () and //,. Let be the 

cost of accepting //■ given that II - is true. That is 

1 J 
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C 00 : The cost of choosing Hq, given that Hq is true. 

C 10 : The cost of choosing H x , given that Hq is true. 

C 01 : The cost of choosing Hq, given that H x is true. 

C xx : The cost of choosing H x , given that H x is true. 

It is reasonable to assume that the associated cost to a correct decision is less than the cost of 
an incorrect decision. That is, c Q0 < c 10 and c xx < c 0) . The average cost can be written as 

C = Z C,/(choosc //, | Hj)P(Hj) 

ij 

= C 00 P(choose H 0 \Hq)P(H 0 ) + C 0X P( choose H 0 \ H X )P(H X ) 

+ C X qP{c\\oosqH 1 \Hq)P{Hq) + C x ^(choose H { \H l )P(H l ). 

Our goal is to find the decision rule such that the average cost is minimized. Show that the 
decision rule can be stated as follows: Choose Hq if and only if 

/y(VI Hq)P{Hq)(C X q - c 00 ) >fj(y I H 1 )P(H 1 )(Cq 1 - C n ) (9.8) 

• Solution 

o First, note that 

/’(choose Hq | Hq) = 1 - /’(choose H x \ H 0 ), 

/’(choose H x | H x ) = 1 - /’(choose Hq \ H x ). 


Therefore, 

C = C 00 [l - /’(choose H x \ Hq)]P(H 0 ) + C 0 ^(choose H 0 \H X )P(H X ) 

+ C X qP{ choose H x \ Hq)P(Hq) + C xx [ 1 - /’(choose H 0 \H X ) ]P(H X ) 

= (C 10 - C 00 )P(choose //, | Hq)P(Hq) + (C 01 - C x 1 )/’(choose H 0 \ H X )P(H X ) 

+ CqqP(Hq) + C xx p(H x ). 


The tenn C 00 p(// 0 ) + C XX P{H X ) is constant (i.e., it does not depend on the 
decision rule). Therefore, to minimize the cost, we need to minimize 

D = P( choose H x | Hq)P(Hq)(C 10 - C 00 ) + /’(choose H 0 \ H X )P(H X )(C 0X - C lx ). 


https://www.probabilitycourse.com/chapter9/9_1_10_solved_probs.php 


9/11 



9/18/2018 


Solved Problems 


The above expression is very similar to the average error probability of the MAP 
test (Eq uation 9.8 ). The only difference is that we have p(/I () )(C l() - C 00 ) instead 

of P{Hq), and we have /t(//| )(C 0 | - C u ) instead of P(JI \). Therefore, we can use 
a decision rule similar to the MAP decision rule. More specifically, we choose H 0 
if and only if 


fy(y \H 0 )P(H 0 )(C m - C 00 ) >/^v|// 1 )P(// 1 )(C 01 - C n ). 


Problem 7 


Let 


X ~ N(0, 4) and Y\X=x ~ N(x, 1). 


Suppose that we have observed the random sample 7 ls V 2 , •••, Y 25 such that the 7/s are i.i.d. 

and have the same distribution as 7. Find a 95% credible interval for X, given that we have 
observed 



Y = 


= 0.56 


n 


Hint: Use the result of Problem 9.2. 


• Solution 

o By part (a) of Problem 9.2. we have 


25(0.56)/1 + 0/4 1 


\ 25/1 + 1/4 ’25/1 + 1/4 

= A(0.5545, 0.0396). 


Therefore, we choose the interval in the form of 


[0.5545 -c, 0.5545 + c]. 


We need to have 
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’(° ' 


P 0.5545 -c<X< 0.5545 +c Y= 0.56 



= 20 


a/o.0396 


- 1 = 0.95 


Solving for c, we obtain 


c = a/O- 03960 ! (0.975) « 0.39 
Therefore, the 95% credible interval for X is 


[0.5545 - 0.39, 0.5545 + 0.39] ~ [0.1645, 0.9445]. 


<— previous 
next —> 
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